Learning to Parse and Ground Natural Language Commands to Robots

This paper describes a weakly supervised approach for understanding natural language commands to robotic systems. Our approach, called the combinatory grounding graph (CGG), takes as input natural language commands paired with groundings and infers the space of parses that best describe how to ground the natural language command. The command is understood in a compositional way, generating a latent hierarchical parse tree that involves relations (such as “to” or “by”) and categories (such as “the elevators” or “the doors”). We show an example parse-grounding tree and show that our system can successfully cluster the meanings of objects and locations.